Evaluating the Wisdom of Crowds in Assessing Phishing Websites

We examine the structure and outcomes of user participation in PhishTank, a phishing-report collator. Anyone who wishes may submit URLs of suspected phishing websites, and may vote on the accuracy of other submissions. We find that PhishTank is dominated by the most active users, and that participation follows a power-law distribution, and that this makes it particularly susceptible to manipulation. We compare PhishTank with a proprietary source of reports, finding PhishTank to be slightly less complete and significantly slower in reaching decisions. We also evaluate the accuracy of PhishTank’s decisions and discuss cases where incorrect information has propagated. We find that users who participate less often are far more likely to make mistakes, and furthermore that users who commit many errors tend to have voted on the same URLs. Finally, we explain how the structure of participation in PhishTank leaves it susceptible to large-scale voting fraud which could undermine its credibility. We also discuss general lessons for leveraging the ‘wisdom of crowds’ in taking security decisions by mass participation.